APP09 Recovery (Compliance Monitoring Application)

APP09 Purpose:

APP09 processes generate test MESSAGING (emulating order sending SITE 1) sending these to either APP01 or APP02 processes to route to APP03s.

APP09 processes receive the order and quote responses for each order generated and report “test success/fail” results as they are received.

Based on testing failures (or successes after failures), APP09 processes may automatically set all MESSAGES for stocks traded by a given APP03 to manual or auto.

By default configuration settings:

APP09DC02 process runs in DC02 and tests all stocks assigned to DC02 MEs, via APP01_APP09DC02 process, running in DC01.

Failures will control quoting conditions in these stocks.

APP09DC01 process runs in DC01 and tests all stocks assigned to DC01 MEs, via APP01_APP09DC01 process, running in DC02.

Failures will control quoting conditions in these stocks.

APP09OMS process runs in DC01 and tests all stocks assigned to DC02 and DC01 MEs, via APP02_APP09OMS process, running in DC02.

Failures will NOT control quoting conditions in these stocks.

Use the following hyperlinks to jump to the desired section of APP09 documentation:

APP09_Recovery_Considerations

APP09_NTM_Control_Commands

APP09_Troubleshooting_Table

APP09_Monitoring_Considerations

APP09 Recovery Considerations:

Stopping/Restart Processes:

- Use NTM Control Utility - Service Control - Process Controller to stop/restart processes.

- Use APP09 nodes only when moving between nodes. (Configuration files utilize specific IP addresses to connect to APP01 and APP02 instances)

- When stopping/restarting APP09 processes:

1) Determine if associated APP01 or APP02 instance used by the affected APP09 instance is also moving hosts.

a. If APP09 associated APP01 or APP02 instances are moving hosts, a different APP09Services.xml configuration file will be required.

The APP09Services.xml files make use of APP09 service level parameters in the maintenance client.

The following APP09 service level parameters are defined for APP09DC01 and APP09DC02 services:

APP01_PRI_PRI_HOST

APP01_PRI_ALT_HOST

APP01_DR_PRI_HOST

APP01_DR_ALT_HOST

The following APP09 service level parameters are defined for the APP09OMS service:

APP02_PRI_PRI_HOST

APP02_PRI_ALT_HOST

APP02_DR_PRI_HOST

APP02_DR_ALT_HOST

b. The APP09Services.xml file can easily be modified to redirect APP09 service being moved to use the correct parameter.

c. There are already some pre-modified versions of APP09Services.xml to aid in these recoveries.

d. If these configuration files are not moved in before the APP09 process restarts, no connections will occur.

2) If the associated APP09 APP01 or APP02 instance is NOT moving to an alternate node, skip to step 3.

a) Move in, or modify the required version of APP09Services.xml file

b) Stop the associated APP01 or APP02 instance on its current node.

c) Copy the day’s APP01 or APP02 fix engine files from d$\\data\ subfolder from current node to same folder on alternate node.

d) Start the associated APP01 or APP02 instance on its alternate node.

e) Open channel for associated APP01 or APP02 instance and confirm channel opens without issue.

3) Stop the APP09 instance.

4) If APP09 process is NOT moving to alternate node, skip to step 5.

a) Copy the day’s APP09 fix engine files from d$\\data\ subfolder from current node to same folder on alternate node.

5) Start APP09 instance.

a) Confirm APP09 testing works without errors.

APP09 testing will automatically start between 5am-3:30pm upon APP09 startup.

APP09 NTM Control Commands :

Auto Quote All MEs:

- Use NTM Control Utility – Service Control - APP09 – Auto Quote All MEs to send all MEs message to set all stocks quote modes to auto.

Auto Quote One ME:

- Use NTM Control Utility – Service Control - APP09 – Auto Quote One ME to send one ME a message to set all stocks quote modes to auto.

Manual Quote All MEs:

- Use NTM Control Utility – Service Control - APP09 – Manual Quote All MEs to send all MEs message to set all stocks quote modes to manual.

Manual Quote One ME:

- Use NTM Control Utility – Service Control - APP09 – Manual Quote One ME to send one ME a message to set all stocks quote modes to manual.

Start All ME Test:

- Use NTM Control Utility – Service Control - APP09 – Start All ME Test to start APP09 testing for all MEs, as configured by stock.

Start One ME Test:

- Use NTM Control Utility – Service Control - APP09 – Start One ME Test to start APP09 testing for a single ME, as configured by stock.

Stop All ME Test:

- Use NTM Control Utility – Service Control - APP09 – Stop All ME Test to stop APP09 testing for all MEs, as configured by stock.

Stop One ME Test:

- Use NTM Control Utility – Service Control - APP09 – Stop One ME Test to stop APP09 testing for a single ME, as configured by stock.

APP09 Troubleshooting Table:

APP09 Symptom	Impacts	Response
APP09 RESTING ORDER QUOTE FAILURES Caused by SITE2 connectivity issues Evidenced by: - APP09 EMT messages indicating RESTING_ORDER_QUOTE_FAILURE. - APP09 Processing Stats monitor indicating RESTING_ORDER_QUOTE_FAILURE. - OMD Stats monitor showing connectivity issues to SITE2 (SITE1 and/or SITE2)	APP09 is not receiving MESSAGES from QRS process as expected. During Trading Hours: cannot fulfill quoting obligations to National Market System. Bookfeed will no longer be updating if QRS process is not connected to SITE2. If 3+ consecutive APP09 failures of any kind, APP09 will send QRS process “set quote to manual” command, causing all affected stocks to be marked as “manual” to SITE2. SITE2 connectivity issues will prevent industry from seeing these “manual” MESSAGES.	1) Refer to: Generalized Event - Cannot Send MESSAGES To SITE2.docx Generalized Recovery Scenario. 2) Refer to: Application Recovery - OMD.docx Application Specific Recoveries.
APP09 RESTING ORDER QUOTE FAILURES NOT caused by SITE2 connectivity issues Evidenced by: - APP09 EMT messages indicating RESTING_ORDER_QUOTE_FAILURE. - APP09 Processing Stats monitor indicating RESTING_ORDER_QUOTE_FAILURE. - OMD Stats monitor showing NO CONNECTIVITY ISSUES to SITE2 (SITE1 and/or SITE2) - APP09_App_Connect_Stats monitor showing possible connectivity problem between APP09 and CQSRI/UQDRI process. - NTM Control MDP CQS/UQDF Display Montage shows MESSAGES marked as “manual”.	APP09 is not receiving MESSAGES from QRS process as expected. During Trading Hours: If 3+ consecutive APP09 failures of any kind, APP09 will send QRS process “set quote to manual” command, causing all affected stocks to be marked as “manual” to SITE2.	1) Confirm scope of impacts. In Stats Monitor: - Are problems specific to: - SITE1 and/or SITE2AQ - DC01 and/or DC02 - Servers? Processes? Channels? 2) Notify management. 3) Work with Tech Services to determine corrective actions. Try to avoid ME stop/restart. - Stop/Restart of APP09 may resolve issue. - Stop/Restart of QRS may resolve issue. - NTM ME option to “Resend ME MESSAGES” may resolve issue. 4) If connectivity cannot be immediately resolved, use NTM Control QRS options by ME to zero MESSAGES or mark all MESSAGES as manual. 5) Consider suspending trading. 6) If QRS restart is tried, once SITE2 connectivity is re-established, QRS will automatically request download of updated MESSAGES from all MEs and send these to SITE2. 7) Once problems are resolved, use NTM ME “Resend ME MESSAGES” option to generate updated MESSAGES from all MEs and send these to SITE2.
APP09 RESTING ORDER EXEC RPT FAILURE WITHOUT APP09 ORDER_EXEC_RPT_FAILURE. Evidenced by: - APP09 EMT messages indicating RESTING_ORDER_EXEC_RPT_FAILURE. - APP09 Processing Stats monitor indicating RESTING_ORDER_EXEC_RPT_FAILURE. - APP09_App_Connect_Stats monitor showing possible connectivity problem between APP09 and FIX.4.1:APP09T process. - NTM Control MDP CQS/UQDF Display Montage shows MESSAGES marked as “manual”. If APP09 is ONLY reporting RESTING_ORDER_EXEC_RPT_FAILURE, without APP09 ORDER_EXEC_RPT_FAILURE, then implication is that the ME is up and processing, and connections to APP09 are likely in tact, but ME APP09 stock may be closed, and need to be re-opened.	APP09 is not receiving ME trade executions from ME, APP01 or APP02 process as expected, BUT is receiving IOC order cancels as expected. During Trading Hours: If 3+ consecutive APP09 failures of any kind, APP09 will send QRS process “set quote to manual” command, causing all affected stocks to be marked as “manual” to SITE2.	1) Confirm scope of impacts. In Stats Monitor: - Are problems specific to: - SITE1 and/or SITE2AQ - DC01 and/or DC02 - Servers? Processes? Channels? 2) Notify management. 3) Work with CSI APP09 order queries and other departments as necessary to determine corrective actions. Try to avoid ME stop/restart. - NTM ME option to “Resume APP09 Issues” may resolve issue. - Stop/Restart of APP01/APP02 may resolve issue; Requires opening of channels. - Stop/Restart of APP09 may resolve issue. 4) If connectivity cannot be immediately resolved, use NTM Control QRS options by ME to zero MESSAGES or mark all MESSAGES as manual. 5) Consider suspending trading. 6) Once problems are resolved, use NTM ME “Resend ME MESSAGES” option to generate updated MESSAGES from all MEs and send these to SITE2.
APP09 RESTING ORDER EXEC RPT FAILURE WITH APP09 ORDER EXEC RPT FAILURE. Evidenced by: - APP09 EMT messages indicating ORDER_EXEC_RPT_FAILURE and RESTING_ORDER_EXEC_RPT FAILURE. - APP09 Processing Stats monitor indicating ORDER_EXEC_RPT_FAILURE and RESTING_ORDER_EXEC_RPT FAILURE. - APP09_App_Connect_Stats monitor showing possible connectivity problem between APP09 and FIX.4.1:APP09T process. - NTM Control MDP CQS/UQDF Display Montage shows MESSAGES marked as “manual”. If APP09 is reporting both RESTING_ORDER_EXEC_RPT_FAILURE and ORDER_EXEC_RPT_FAILURE simultaneously, then likely issue is caused by ME being down, or ME connection to APP09 APP02/APP01 is broken, or APP09 APP02/APP01 connection to APP09 is broken.	APP09 is not receiving ME trade executions from ME, APP01 or APP02 process as expected, NOR receiving IOC order cancels as expected. During Trading Hours: If 3+ consecutive APP09 failures of any kind, APP09 will send QRS process “set quote to manual” command, causing all affected stocks to be marked as “manual” to SITE2.	1) Confirm scope of impacts. In Stats Monitor: - Are problems specific to: - SITE1 and/or SITE2AQ - DC01 and/or DC02 - Servers? Processes? Channels? 2) Notify management. 3) Work with CSI APP09 order queries and other departments as necessary to determine corrective actions. Try to avoid ME stop/restart. - Stop/Restart of APP01/APP02 may resolve issue; Requires opening of channels. - Stop/Restart of APP09 may resolve issue. - NTM ME option to “Resume APP09 Issues” may resolve issue. 4) If connectivity cannot be immediately resolved, use NTM Control QRS options by ME to zero MESSAGES or mark all MESSAGES as manual. 5) Consider suspending trading. 6) Once problems are resolved, use NTM ME “Resend ME MESSAGES” option to generate updated MESSAGES from all MEs and send these to SITE2.

APP09 Symptom

Impacts

Response

APP09 RESTING ORDER QUOTE FAILURES

Caused by SITE2 connectivity issues

Evidenced by:

- APP09 EMT messages indicating RESTING_ORDER_QUOTE_FAILURE.

- APP09 Processing Stats monitor indicating RESTING_ORDER_QUOTE_FAILURE.

- OMD Stats monitor showing connectivity issues to SITE2 (SITE1 and/or SITE2)

APP09 is not receiving MESSAGES from QRS process as expected.

During Trading Hours:

cannot fulfill quoting obligations to National Market System.

Bookfeed will no longer be updating if QRS process is not connected to SITE2.

If 3+ consecutive APP09 failures of any kind, APP09 will send QRS process “set quote to manual” command, causing all affected stocks to be marked as “manual” to SITE2.

SITE2 connectivity issues will prevent industry from seeing these “manual” MESSAGES.

1) Refer to:

Generalized Event - Cannot Send MESSAGES To SITE2.docx

Generalized Recovery Scenario.

2) Refer to:

Application Recovery - OMD.docx

Application Specific Recoveries.

APP09 RESTING ORDER QUOTE FAILURES

NOT caused by SITE2 connectivity issues

Evidenced by:

- APP09 EMT messages indicating RESTING_ORDER_QUOTE_FAILURE.

- APP09 Processing Stats monitor indicating RESTING_ORDER_QUOTE_FAILURE.

- OMD Stats monitor showing NO CONNECTIVITY ISSUES to SITE2 (SITE1 and/or SITE2)

- APP09_App_Connect_Stats monitor showing possible connectivity problem between APP09 and CQSRI/UQDRI process.

- NTM Control MDP CQS/UQDF Display Montage shows MESSAGES marked as “manual”.

APP09 is not receiving MESSAGES from QRS process as expected.

During Trading Hours:

If 3+ consecutive APP09 failures of any kind, APP09 will send QRS process “set quote to manual” command, causing all affected stocks to be marked as “manual” to SITE2.

1) Confirm scope of impacts.

In Stats Monitor:

- Are problems specific to:

- SITE1 and/or SITE2AQ

- DC01 and/or DC02

- Servers? Processes? Channels?

2) Notify management.

3) Work with Tech Services to determine corrective actions. Try to avoid ME stop/restart.

- Stop/Restart of APP09 may resolve issue.

- Stop/Restart of QRS may resolve issue.

- NTM ME option to “Resend ME MESSAGES” may resolve issue.

4) If connectivity cannot be immediately resolved, use NTM Control QRS options by ME to zero MESSAGES or mark all MESSAGES as manual.

5) Consider suspending trading.

6) If QRS restart is tried, once SITE2 connectivity is re-established, QRS will automatically request download of updated MESSAGES from all MEs and send these to SITE2.

7) Once problems are resolved, use NTM ME “Resend ME MESSAGES” option to generate updated MESSAGES from all MEs and send these to SITE2.

APP09 RESTING ORDER EXEC RPT FAILURE

WITHOUT APP09 ORDER_EXEC_RPT_FAILURE.

Evidenced by:

- APP09 EMT messages indicating RESTING_ORDER_EXEC_RPT_FAILURE.

- APP09 Processing Stats monitor indicating RESTING_ORDER_EXEC_RPT_FAILURE.

- APP09_App_Connect_Stats monitor showing possible connectivity problem between APP09 and FIX.4.1:APP09T process.

- NTM Control MDP CQS/UQDF Display Montage shows MESSAGES marked as “manual”.

If APP09 is ONLY reporting RESTING_ORDER_EXEC_RPT_FAILURE, without APP09 ORDER_EXEC_RPT_FAILURE, then implication is that the ME is up and processing, and connections to APP09 are likely in tact, but ME APP09 stock may be closed, and need to be re-opened.

APP09 is not receiving ME trade executions from ME, APP01 or APP02 process as expected, BUT is receiving IOC order cancels as expected.

During Trading Hours:

If 3+ consecutive APP09 failures of any kind, APP09 will send QRS process “set quote to manual” command, causing all affected stocks to be marked as “manual” to SITE2.

1) Confirm scope of impacts.

In Stats Monitor:

- Are problems specific to:

- SITE1 and/or SITE2AQ

- DC01 and/or DC02

- Servers? Processes? Channels?

2) Notify management.

3) Work with CSI APP09 order queries and other departments as necessary to determine corrective actions. Try to avoid ME stop/restart.

- NTM ME option to “Resume APP09 Issues” may resolve issue.

- Stop/Restart of APP01/APP02 may resolve issue; Requires opening of channels.

- Stop/Restart of APP09 may resolve issue.

4) If connectivity cannot be immediately resolved, use NTM Control QRS options by ME to zero MESSAGES or mark all MESSAGES as manual.

5) Consider suspending trading.

6) Once problems are resolved, use NTM ME “Resend ME MESSAGES” option to generate updated MESSAGES from all MEs and send these to SITE2.

APP09 RESTING ORDER EXEC RPT FAILURE

WITH APP09 ORDER EXEC RPT FAILURE.

Evidenced by:

- APP09 EMT messages indicating ORDER_EXEC_RPT_FAILURE and RESTING_ORDER_EXEC_RPT FAILURE.

- APP09 Processing Stats monitor indicating ORDER_EXEC_RPT_FAILURE and RESTING_ORDER_EXEC_RPT FAILURE.

- APP09_App_Connect_Stats monitor showing possible connectivity problem between APP09 and FIX.4.1:APP09T process.

- NTM Control MDP CQS/UQDF Display Montage shows MESSAGES marked as “manual”.

If APP09 is reporting both RESTING_ORDER_EXEC_RPT_FAILURE and ORDER_EXEC_RPT_FAILURE simultaneously, then likely issue is caused by ME being down, or ME connection to APP09 APP02/APP01 is broken, or APP09 APP02/APP01 connection to APP09 is broken.

APP09 is not receiving ME trade executions from ME, APP01 or APP02 process as expected, NOR receiving IOC order cancels as expected.

During Trading Hours:

If 3+ consecutive APP09 failures of any kind, APP09 will send QRS process “set quote to manual” command, causing all affected stocks to be marked as “manual” to SITE2.

1) Confirm scope of impacts.

In Stats Monitor:

- Are problems specific to:

- SITE1 and/or SITE2AQ

- DC01 and/or DC02

- Servers? Processes? Channels?

2) Notify management.

3) Work with CSI APP09 order queries and other departments as necessary to determine corrective actions. Try to avoid ME stop/restart.

- Stop/Restart of APP01/APP02 may resolve issue; Requires opening of channels.

- Stop/Restart of APP09 may resolve issue.

- NTM ME option to “Resume APP09 Issues” may resolve issue.

4) If connectivity cannot be immediately resolved, use NTM Control QRS options by ME to zero MESSAGES or mark all MESSAGES as manual.

5) Consider suspending trading.

6) Once problems are resolved, use NTM ME “Resend ME MESSAGES” option to generate updated MESSAGES from all MEs and send these to SITE2.

APP09 Monitoring Considerations:

Stats Monitors: APP09 App Connect Stats	To Start:	Key Indicators to Monitor:	Symptom:	Response:
Processes generate test MESSAGING and cancels to APP03s via APP01 or APP02 processes and report communication failures from APP03s and QRS processes. Monitor shows connection status between APP09 fix engine and APP01, APP02 and QRS processes.	PROD MENU: Trading Applications Monitoring Menu To Exit: Close Window	- Color of data in columns - Name, - Status, - Write Queue	Data is RED.	Process is either down or multicast data is not being received by monitor. 1) Check status of process 2) If process is up, call Technical Services.
			Status is Created, Disconnected or Open	APP01, APP02 or QRS is not connected. 1) Use NTM Control Utility APP01 or APP02 Service Controls to Open Channels. 2) Stop/Restart APP09 or QRS process if commonality between disconnected statuses can be detected. 3) Call Production Control if needed.
			Write Queue is non-zero values and not reducing as expected.	Messages cannot be sent from source to destination unless IPC channel is connected. 1) Stop/Restart destination process so as not to accidentally delete queued messages; Do not stop/restart source process. 2) Notify Production Support if issues.

Stats Monitors: APP09 App Processing Stats	To Start:	Key Indicators to Monitor:	Symptom:	Response:
Processes generate test MESSAGING and cancels to APP03s via APP01 or APP02 processes and report communication failures from APP03s and QRS processes. Monitor shows testing status and processing statistics of APP09 processes.	PROD MENU: Trading Applications Monitoring Menu To Exit: Close Window	- Color of data in columns - Service, - ME Names, - Status, - Quote Mode, - Test Result, - TestDesp	Data is RED. Process is either down or multicast data is not being received by monitor.	1) Check status of process 2) If process is up, call Technical Services.
			Status is not Testing APP09 testing has not been enabled.	1) Use NTM Control Utility APP09 Service Controls to control testing. 2) Stop/Restart APP09 process if process isn't responding as expected. 3) Call Production Control if needed.
			Quote Mode is Manual APP09 test cycles have failed 3 consecutive times and not succeeded in 3 subsequent consecutive times.	1) If failure reasons are quote related, check QRS processes and processing. 2) If failures are order or execution report related, check ME processing via statistics monitors and CSI APP09 order queries if necessary. 3) Call Production Control if needed.
			Test Results are not PASS and/or TestDesp are not successful. APP09 test cycles have failed 3 consecutive times and not succeeded in 3 subsequent consecutive times.	1) If failure reasons are quote related, check QRS processes and processing. 2) If failures are order or execution report related, check ME processing via statistics monitors and CSI APP09 order queries if necessary. 3) Call Production Control if needed.